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The purpose of this study was to gain an understanding of what e-Learning executives believe are the 
critical success factors for companies to successfully deliver training and education over the world-wide 
web. The study was a qualitative, multiple-case study design including in-depth, semi-structured interviews 
that incorporated verbal critical incident examples; analysis of extensive forms and observations; and it 
incorporated a validated sun’ey tool, the Dimensions of the Learning Organization Questionnaire (DLOQ) 
(Watkins and Marsick, 2003 ). 
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e-Learning technology is evolving at incredible speeds and transforming the way companies are delivering learning 
to their employees, vendors, and customers. According to Allison Rossett (2002), in the year 2000, most authors 
would have been overly optimistic and enthusiastic about the benefits of e-Learning, and yet a year or two later, 
after some experience, the same authors are more cautious, acknowledging challenges, having observed e-Learning 
investments go awry, and they recognize the need for sound strategies to increase the chances of success. Rosenberg 
(2001) suggests that today it is no longer a question of “whether" organizations will implement e-Leaming, but a 
question of “whether they will do it well” (p. xvi). He suggests that success requires an effective e-Learning strategy 
that addresses both design and technology issues as well as acceptance and support issues. 

Today, in part due to its relative infancy (having started in the mid-1990s) and in part due to the speed of 
technological change, there is a dearth of research available for understanding what executives responsible for e- 
Learning delivery believe to be the most important factors necessary to effectively deliver e-Learning in their 
organizations. Specifically, little is known about what e-Learning leaders in the field perceive to be the important 
principles to follow to ensure that the e-Learning programs they are responsible for delivering are successful. There 
is a growing body of theory surrounding the issues associated with e-Learning (Rosenberg, 2001; Rossett, 2002). 
However, there is little in the way of comprehensive research available that either supports or contradicts the current 
thinking. 

Based on a review of the literature and identification of themes, the four factors that needed to be researched 
included: (1) the type and amount of executive, direct management and other support present (Berkhard & Pritchard, 
1992; Burke, 2002; Cross & Dublin, 2002; and Rosenberg, 2001); (2) how closely the organization’s culture 
supports individual, group, or organizational learning (Bruner, 1996, Rosenberg, 2001; Vygotsky, 1978; and 
Watkins and Marsick, 2003); (3) the external motivators the organization puts in place to encourage learner 
participation in the program and/or learner completion of the program, and how these motivators may ignite intrinsic 
motivators (Dick and Carey, 2001; Dweck, 1999; Esque and McClausland, 1997; Hadre, 2003; Gagne, 1992, Mauer, 
2002; Moshinskie, 2002; Rossett, 2002 and Wiltscher, 1999); and (4) the relevance of the content to the learner’s 
current job, current job tasks, and future positions (Mauer, 2002; Moshinskie, 2002; and Schank, 2005). 

Therefore, the problem this study responded to is an effort to fill the gap between what is theorized about 
concerning critical success factors for delivering e-Learning programs in corporations and what e-Learning 
executives actually believe occurs in practice. These issues surfaced in the few comprehensive works on e-Learning 
success factors found in the literature (Rosenberg, 2001; Rossett, 2002) and in the researcher’s experience 
(Armstrong, 2007). 

E-Learning Conceptual Framework 

Over the past 25 years, the researcher has worked in corporate America designing training programs and over the 
past decade has been working in the field of web-based learning. During this time, the researcher has collected 

Copyright © 2008 Ann W. Armstrong 



20-1 




numerous critical incidents from in excess of 15 Fortune 500 companies, covering over 25 major web-based learning 
initiatives, and has been analyzing these incidents for commonality and themes. In addition, the researcher has 
managed projects ranging in size from $60,000 to $4,500,000 and rollouts of web-based learning to 15,000 
employees in 8 weeks and to 120,000 employees over a six-month period. Archival data, which were reviewed, 
include notes from 26 semi-structured interviews in the spring and summer of 2001, minutes from 6 e-Learning 
Executive Roundtables in 2003, 2004, and 2005, and 41 presentations from the Virtual University: Corporate and 
Academic held at Teachers College Columbia University in June 2002, and the results from a pilot study conducted 
for this study the winter of 2005. From these experiences — combined with themes covered in this literature review 
around culture, support, motivation, relevant content, and outcomes — the researcher constructed an e-Learning 
Conceptual Framework to guide the study. 





Figure 1. e-learning conceptual framework. 

The conceptual framework guided the study and provided a basis for the assumptions: (1) Support, (2) Culture, 
(3) External Learner Motivation Factors, (4) Program Design and Content, and (5) Outcomes (Kirkpatrick, 1998). 
The Support, Learner Motivation Factors, Program Design, and Content and Outcomes variables were synthesized 
against three data sources: (1) case descriptions, (2) critical incidents, and (3) critical success factor question 
answers; the Culture variable was examined against three data sources: (1) case descriptions, (2) DLOQ e-Learning 
Leader Mean score from the sample participants, and (3) DLOQ answers from the participants. 

Research Questions and Purpose Statement 



The purpose of this study was to gain an understanding of what e-Learning executives (e.g.. Chief Learning 
Officers, Senior Vice Presidents, and Vice Presidents of Learning whose responsibilities include design and delivery 
of learning over the world-wide web) learned from the experience of defining, designing, developing, and deploying 
large-scale web-based learning programs for adult professional development in major corporations. Additionally, it 
sought to uncover what factors these executives believe to be critical and necessary for the organization to 
successfully deliver training and education over the world-wide web. 

This study drew on a prior study conducted by the researcher to uncover what e-Learning leaders were doing as 
they implemented e-Learning in their corporations. In the current study, the researcher desired to understand what 
these e-Learning leaders had learned as they implemented e-Learning programs in their companies so that this 
documented information could be made available to others in the field as they launched new e-Learning programs. 
The core research questions were: 

1. What perceptions do e-Learning leaders have regarding the way different forms of support contribute to 
the success or failure of online learning program delivery? Support such as the following: 

Executive Line of Business sponsorship 
Executive e-Learning leadership 
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Direct manager involvement 

Participation in and recognition by a Community of Practice 
Technology infrastructure 

2. What perceptions do e-Learning executives hold concerning the impact on e-Leaming programs of specific 

aspects of a learning culture (e.g., how are external incentives, rewards, and other learner motivational 
strategies perceived and used by the e-Learning executive in the culture to influence learner participation 
in a program, etc.)? 

3. How important do e-Learning leaders believe is the nature of the content and its relevance to the learner’s 
job in driving motivation? 

To address the research questions, the study focused on fully understanding the role e-Learning practitioners 
play in defining, designing, developing, and deploying large scale web-based learning and blended learning 
solutions for professional development of their organization’s employees. “ Blended learning” is a mixture of web- 
based learning and more traditional methods of learning, such as instructor-led classroom, seminars, and workshops. 
Specifically, the study sought to identify what are the critical success factors that are required to be operating for 
these programs to be successful. The study sought to go beyond the theories currently emerging in much of the 
literature by uncovering how the e-Learning executives experience these successes and/or failures in practice 
(Armstrong, 2007). 

Methodology with Limitations 

The context of this study was e-Learning executives in some of the largest U.S. corporations across a variety of 
industries. The researcher secured interviews from 2 executives from the world’s top three Banks, 2 from two of 
the world's largest investment advisory firms, 1 from one of the world's premier Construction companies, 1 from 
one of the world's largest insurance companies, and 1 from one of the world’s top pharmaceutical companies. 

Through the use of a qualitative, multi-person, mixed method case study, this study explored the e-Learning 
phenomenon by using a variety of data sources, including: a Pre-Interview Data Form (PIDF); participant answers to 
the Dimensions of the Learning Organization Questionnaire (DLOQ); semi-structured interviews, including critical 
e-Leaming events; field notes, including memos, meeting minutes, and researcher reflections; and case documents 
(Armstrong, 2007). 

The purpose of the interviews of the seven e-Leaming leaders in large corporations was to identify and discover 
what these practitioners perceived to be important tenets that, when observed and followed, can assist program 
sponsors and designers in ensuring that program goals and objectives for location-independent online learning are 
met. e-Learning leaders interviewed were asked to complete Watkins and Marsick's Dimensions of the Learning 
Organization (DLOQ) questionnaire. 

Next this study asked the seven e-Learning practitioners, the research participants, to think critically about, and 
reflect on (Flanagan, 1954), their experience of delivering large-scale rollouts of web-based learning programs in 
their respective organizations. They were asked specifically to identify and describe the elements they believed have 
contributed to a program’s success or failure, as perceived by them, with respect to support, culture, external learner 
motivation strategies, and content. Critical incidents surrounding their best e-Learning programs and their worst 
were described by each participant. As they answered these questions, information was sorted into the following 
major categories that supported answering the research questions: support, culture, motivation, program design, and 
content. 

The study allowed the researcher to search for deeper meanings and understanding of the role played by 
support, corporate culture, external learner motivation strategies, and content as these are perceived by e-Learning 
leaders in the organization to impact web-based learning programs. The study sought to identify other critical 
success factors that are perceived by the practitioners to play a role in e-Learning design. The study has several 
limitations (Armstrong, 2007): (1) The researcher's knowledge and experience with some of the participants may 
have made them more comfortable in some ways and yet guarded in others; (2) case studies by their nature present a 
number of limitations for research. This study has been designed to minimize these limitations while recognizing 
their inherent existence. A limited number of case participants where each participant has a single interview will be 
subject to retrospective recall where the practitioners may have had selected memory based on their own biases and 
beliefs; (3) there is no objective verification of the participants’ retrospective recall or views; (4) the small sample 
size of this study prohibits generalization of the findings to the overall population of corporate e-Learning leaders. 
Since the study only involves seven e-Learning leaders, only a small sample of leaders that deliver web-based 
learning programs was studied; and (5) having been submerged in the field for over three decades, the researcher 
brought experience, beliefs, and biases to the process. This knowledge and experience was at times a benefit and at 
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others could have been detrimental. To safeguard against negative bias, the researcher had two colleagues code the 
data, review the list of codes, and help to identify emerging themes and patterns. 

Results and Findings 

Table 1 shows a synthesis of the success factors among all data sources except the DLOQ with emphasis on the 
factors that received either unanimity across data sources or that the researcher ranked as strong or medium-strong 
across data sources. The two indicators “Blended ILT and Web-based” and “Blended OJT, Coaching with Web- 
based,” for purpose of the analysis, were aggregated as “Blended Traditional and Web-based” to reflect that the 
program was a blend of web delivery with some other more traditional method. To perform this analysis, the matrix 
was created that incorporated the indicators from the categories of Support, Motivation, Program Design, and 
Outcomes and listed them in descending order starting with the results on the case description data source. Strong 
indicators are in dark gray and shown with double asterisks; medium indicators in light gray with single asterisks; 
other indicators are shown with white background. Strength of indicators is based on frequency of occurrence in the 
data and may or may not indicate the relative importance in determining success, as some indicators; for example, 
“Community of Practice,” are only evidenced in organizations that are highly experienced and on the leading edge 
in delivering online programs. 

The study’s findings confirmed the importance of the five analytic categories and their 30 indicators. What 
emerged was a theoretical framework, eSUCCESS. The eSUCCESS framework represents a fusion of theory and 
practice by transforming the experiences and knowledge of a select group of e-Learning leaders into eight tenets that 
can be applied by others for future action and that will assist them in garnering positive results. The tenets are: 

(1) Executive Sponsorship 

(2) Support from the Organization 

(3) Understand and Motivate the Learner 

(4) Culture Fosters Learning 

(5) Content is Relevant to Learner & Organization 

(6) Evaluate and Assess 

(7) Structure of Program is Engaging, Interactive and Blended 

(8) Simulate the Work Environment and Work Tasks 

What did not emerge from this study was a definitive set of rules that can be applied by e-Learning leaders in all 
situations. Instead of recommending a finite list of success factors, based on this study’s findings, the proposed 
research-based framework in Figure 2 focuses on helping practitioners understand the eight tenets, which, when 
followed in ways that work in their environment, will help them ensure a successful e-Learning implementation in 
their organization. 
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Table 1. Synthesis of Indicators Ranked — All Data Sources except DLOQ 



Key Experiences: 

Support, Motivation, Program Design, 
Outcomes 


Case Description 
n=7 


Critical Incident 
n=7 


Critical Success Factors 
n=7 


Total 


% 


Total 


% 


Total 


% 


Executive Sponsorship** 


7 


100 


7 


100 


6 


85.7 


Relevance of Content/Certification** 


7 


100 


7 


100 


7 


100 


Content Culture Specific/Purposeful** 


7 


100 


7 


100 


7 


100 


High Quality, Engaging, Enjoyable** 


7 


100 


7 


100 


5 


71.4 


Case Study** 


7 


100 


7 


100 


1 


14.3 


Blended Traditional and Web-based** 


7 


100 


7 


100 


0 


0 


Kirkpatrick Level 1 -Reaction** 


7 


100 


0 


0 


0 


0 


Kirkpatrick Level 2-Learning** 


7 


100 


0 


0 


0 


0 


Executive e-Learning Leadership** 


7 


100 


3 


42.9 


3 


42.9 


Executive Line of Business** 


7 


100 


3 


42.9 


3 


42.9 


Information Technology Leader** 


7 


100 


3 


42.9 


0 


0 


Direct Manager Involvement* 


5 


71.4 


1 


14.3 


1 


14.3 


Improved Confidence* 


5 


71.4 


0 


0 


0 


0 


Benefit Communicated Executive* 


5 


71.4 


1 


14.3 


4 


57.1 


Part of Performance Plan* 


5 


71.4 


1 


14.3 


0 


0 


Manager Bonus 


4 


57.1 


1 


14.3 


0 


0 


Course Mandated Outside Regulators 


4 


57.1 


4 


57.1 


1 


14.3 


Rewarded Additional Resources 


4 


57.1 


4 


57.1 


1 


14.3 


Built into the Organization 


4 


57.1 


2 


28.6 


0 


0 


Technology Platform* 


3 


42.9 


5 


71.4 


3 


42.9 


Receive a Punishment 


3 


42.9 


1 


14.3 


0 


0 


Peer Pressure 


3 


42.9 


2 


28.6 


1 


14.3 


Loss of Job 


3 


42.9 


2 


28.6 


1 


14.3 


Other Positive Incentives 


3 


42.9 


2 


28.6 


1 


14.3 


Simulation 


3 


42.9 


2 


28.6 


3 


42.9 


Community of Practice 


3 


42.9 


0 


0 


0 


0 


Reputable Subject Matter Expert 


2 


28.6 


2 


28.6 


3 


42.9 


Good Instructional Design 


2 


28.6 


1 


14.3 


3 


42.9 


Branded to the Culture 


2 


28.6 


2 


28.6 


1 


14.3 


Continuing Education Requirement 


2 


28.6 


1 


14.3 


0 


0 


Strong Indicator -Dark Gray** Medium-Strong Indicator-Light-Medium Gray* 
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Building Blocks for ^-SUCCESS 





/ z 


7 






Structure 

Blended ILT 
OJT 

Coaching 

Engaging Innovative 
Visual Auditory 


SIMULATE THE 
WORKPLACE 

Case Studies 
“Caselettes” 
Branded TV Culture 


/ 


EVALUATE AND ASSESS 
Built into Courseware 
Passing Score Required 
Kirkpatrick Levels 1,2, 3, 4 


/ 


/ 




/ 


Support 

eLearning Leadership 
IT Leadership 
Line of Business 
Direct Manager 
IT Infrastructure 


UNDERSTAND AND 
MOTIVATE THE 
LEARNER 

Relevant Content 
Certification 
Benefit Communicated 
Performance Plan 
“Peer Pressure” 


CULTURE 

Continuous Learning 
Systems to Share Learning 
Collaboration and Team 
Learning 

Inquiry ana Dialogue 
Collective Vision 
Connecting to Environment 


CONTENT 

Relevant to Job 
Specific to Culture 
Purposeful 

High Quality, Enjoying 
Blended 
Case Studies 
Reputable SME 


7 


Executive Sponsorship 
“Built into the Organization” 



Figure 2. Building blocks for e-SUCCESS. 

Conclusions and Recommendations 

CONCLUSION 1: Gamering executive sponsorship in the form of time, money, and communications is a key factor 
in driving successful e-Learning initiatives. 

Research question 1 is focused on the perception e-Leaming leaders have regarding different types of support 
that contribute to the success or failure of online learning program delivery. Across all three data sources, the 
research participants identified Executive Sponsorship as a strong requirement for success that is consistent with the 
literature (Cross & Dublin, 2002; Moshinskie, 2003 and Rosenberg, 2001). Executive e-Learning leadership and 
executive line of business leadership also received high rankings, being cited in all 7 (or 100%) of the case 
descriptions and 3 out of the 7 (42.9%) in both the critical incidents and the critical success factors question. 
Similarly, Information Technology Leadership was mentioned in 7 out of the 7 (100%) case descriptions and in 3 
out of 7 (42.9%) of the critical incidents, but in none of the critical success factors. Another indicator with 
interesting findings is the Technology Platform indicator. The indicator was cited in 5 out of the 7 (71.4%) critical 
incidents and in 3 of the 7 (42.9%) in both the case descriptions and the critical success factors. The indicator was 
discussed in the critical incidents when the study participants were discussing the “low point” in their e-Learning 
experiences, and in 5 out of the 7 (71.4%) cases, that “low point” occurred when the technology failed in some way, 
usually due to lack of adequate bandwidth. The lack of a “strong,” “robust,” “reliable” platform was mentioned in 5 
out of the 7 (71.4%) critical incidents as the factor that caused the e-Learning program to either fail or have 
significant problems during the rollout. It appears that when technology fails, it is a showstopper because, if not 
reliable, the show cannot go on; but that alone will not make the course that is being delivered via the technology 
and supported in the organization a success. (Armstrong, 2007). 

Recommendation 1. The researcher recommends that e-Learning leaders strive to seek a place at the executive 
table with the CEO and others in the “C-Suite.” Or if the e-Learning programs are owned within a Line of Business, 
that the e-Learning leaders report to the LOB leader who reports directly to the CEO. In addition, e-Learning leaders 
must have a robust and reliable e-Leaming platform. 

CONCLUSION 2: Having a culture that supports and fosters learning is a key factor in driving successful e- 
Learning initiatives. 

Research question 2 is focused on the perceptions e-Learning leaders hold concerning specific aspects of culture 
and their impact on e-Learning program success or failure. The respondents in this study were aligned with the 
literature on organizational learning and the core patterns needed for a learning organization as identified at the 
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individual, team, and organizational level (Watkins & Marsick, 2003), which include: creation of continuous 
learning opportunities, promotion of inquiry and dialogue, encouragement of collaboration and team learning, 
development of systems to capture and share knowledge, creation of a collective vision and empowering individuals 
toward that vision, connection of the organization to its environment and development of leaders who model and 
sponsor learning. The mean score of the Study Participants was high in each of the dimensions. Having the 
organizations scoring high on each of the dimensions of the learning organization categories is likely to enhance an 
e-Leaming leader's chance of successfully delivering an e-Learning program, which leads to the researcher’s 
following recommendation: 

Recommendation 1. The researcher recommends that organizations have both their e-Learning leaders and 
members of the broader employee pool take the DLOQ and review their scores against those of the e-Learning 
leaders before embarking on an e-Learning initiative. If their score in any area is low, they should endeavor to 
improve that area before launching the e-Learning initiative. Or at a minimum, recognize that the lower score 
presents a risk to their success. 

CONCLUSION 3: Learner motivation, particularly strategies that ignite intrinsic motivation, is a key to success for 
e-Leaming programs. 

Feedback from the participants points to the importance of learner motivation in the successful delivery of e- 
Learning programs and corroborates the literature in this regard. e-Learning leaders have used a variety of external 
motivators to ignite internal learner motivation. The most successful motivators were programs that led to 
certifications, accreditations, and internal designations. Given that these types of motivators in all probability tapped 
into some internal motivators, such as enhancing self esteem and self image, it may be a benefit for e-Learning 
leaders to test their learners using self-scoring surveys prior to having them enroll in an e-Leaming program to 
determine what types of external motivators need to be put into place for the program to succeed. In addition, 
designers must ensure that the program is designed in such a manner that the learners understand how the content is 
relevant to them personally and to their current or future job. 

Recommendation 1. The researcher recommends that organizations test their target audience in a effort to 
understand whether their internal motivation to learn is high and whether they are in a state of constant learning or 
not. 

Recommendation 2. The researcher recommends that designers of e-Learning ensure that the target audience 
understands how the e-Learning program is relevant to the self and can visualize themselves in a desired future state, 
which results from successful completion of the e-Learning. 

CONCLUSION 4: Program design that is culture specific, purposeful to the job, engaging, high quality, interactive, 
incorporates a case study, and is blended with more traditional educational methods is a key to success for e- 
Learning programs. 

In response to research question 3, all of the participants, 7 out of 7 (100%), mentioned in their critical incident 
experiences the importance of designing a program that was specific to the corporate culture, had relevance to the 
current or future job and its tasks, was high quality, engaging, and interactive, and was based on a real life 
experience. All of these program attributes were expected and found in the literature. Moshinskie (2002) points out 
that it is especially important that e-Learning capture the essence of the learner’s job situation and specifically 
address tasks they will have to perform on the job. Mauer (2002), through the use of ELDO, suggests a linkage 
between the learner’s current self and possible future self and shows that work and job context influence the 
person's desire to participate in and complete the learning program, and Schank (2005) encourages designers to 
work with content that, through goal-based scenarios, simulates the learner’s work environment and work tasks. 

New findings in this study reveal that blending the e-Learning with more traditional methods is likely to lead to 
a successful program. Seven out of 7 (100%) of the e-Learning leaders cited some form of a blended solution when 
describing their most successful e-Learning event. And importantly, Allison Rossett (2006), in her presentation at 
the CLO Conference in October 2006, discussed blending its importance that her research is uncovering. 

Recommendation 1. The researcher recommends that instructional designers pay close attention to cultural 
elements; that they ensure the content is relevant to the learners’ current and/or future job; that the learners 
understand how the program can benefit them and have the opportunity to visualize themselves in the new job or 
role; that content be engaging, interactive, and that the e-Learning program be blended with one or more traditional 
ways of teaching. 

Significance for Practice 

Companies, to stay competitive, are forced to select, implement, and rollout new learning technologies in an 
atmosphere of constant change and innovation often without all the necessary information to determine the real 
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benefits to the organization. This study provides the executive sponsors and managers in these organizations with a 

research-based framework on which to base these decisions. 
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